{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeadDkMiISin"
      },
      "source": [
        "# Gemini API: Safety Quickstart"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lEXQ3OwKIa-O"
      },
      "source": [
        "<table align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/google-gemini/cookbook/blob/main/quickstarts/Safety.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uOxMUKTxR-_j"
      },
      "source": [
        "The Gemini API has adjustable safety settings. This notebook walks you through how to use them. You'll write a prompt that's blocked, see the reason why, and then adjust the filters to unblock it.\n",
        "\n",
        "Safety is an important topic, and you can learn more with the links at the end of this notebook. Here, we're focused on the code."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gHYFrFPjSGNq"
      },
      "source": [
        "## Set up your API key\n",
        "\n",
        "If you want to quickly try out the Gemini API, you can use `curl` commands to call the methods in the REST API.\n",
        "\n",
        "This notebook contains `curl` commands you can run in Google Colab, or copy to your terminal.\n",
        "\n",
        "To run this notebook, your API key must be stored it in a Colab Secret named GOOGLE_API_KEY. If you are running in a different environment, you can store your key in an environment variable. See [Authentication](https://github.com/google-gemini/cookbook/blob/main/quickstarts/Authentication.ipynb) to learn more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "405ee147f509"
      },
      "outputs": [],
      "source": [
        "!apt install jq"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ab9ASynfcIZn"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "from google.colab import userdata"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7b547b1d5cad"
      },
      "outputs": [],
      "source": [
        "os.environ['GOOGLE_API_KEY'] = userdata.get('GOOGLE_API_KEY')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 108,
      "metadata": {
        "id": "3defec89594e"
      },
      "outputs": [],
      "source": [
        "os.environ['UNSAFE_PROMPT'] = "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LZfoK3I3hu6V"
      },
      "source": [
        "## Prompt Feedback\n",
        "\n",
        "The result returned by the [Model.generate_content](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content) method is a [genai.GenerateContentResponse](https://ai.google.dev/api/python/google/generativeai/types/GenerateContentResponse)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 109,
      "metadata": {
        "id": "6d9e5d84541c"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "echo '{\n",
        "      \"contents\": [{\n",
        "        \"parts\":[{\n",
        "          \"text\": \"'$UNSAFE_PROMPT'\"}]}]}' > request.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "id": "2bcfnGEviwTI"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"promptFeedback\": {\n",
            "    \"blockReason\": \"SAFETY\",\n",
            "    \"safetyRatings\": [\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "        \"probability\": \"MEDIUM\"\n",
            "      },\n",
            "      {\n",
            "        \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "        \"probability\": \"NEGLIGIBLE\"\n",
            "      }\n",
            "    ]\n",
            "  }\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "\n",
        "curl https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=$GOOGLE_API_KEY \\\n",
        "    -H 'Content-Type: application/json' \\\n",
        "    -X POST \\\n",
        "    -d @request.json  2> /dev/null | tee response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WR_2A_sxk8sK"
      },
      "source": [
        "Above you can see that the response object gives you safety feedback about the prompt in two ways:\n",
        "\n",
        "* The `prompt_feedback.safety_ratings` attribute contains a list of safety ratings for the input prompt.\n",
        "* If your prompt is blocked, `prompt_feedback.block_reason` field will explain why."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "72b4a8808bb9"
      },
      "source": [
        "If the prompt is blocked because of the safety ratings, you will not get any candidates in the response."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4672af98ac57"
      },
      "source": [
        "### Safety settings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a6229f6d3a1"
      },
      "source": [
        "Adjust the safety settings and the prompt is no longer blocked:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 134,
      "metadata": {
        "id": "9c38561789c2"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "echo '{\n",
        "    \"safetySettings\": [\n",
        "        {'category': 7, 'threshold': 4}\n",
        "    ],\n",
        "    \"contents\": [{\n",
        "        \"parts\":[{\n",
        "          \"text\": \"'$UNSAFE_PROMPT'\"}]}]}' > request.json"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 143,
      "metadata": {
        "id": "338fb9a6af78"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "\n",
        "curl https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=$GOOGLE_API_KEY \\\n",
        "    -H 'Content-Type: application/json' \\\n",
        "    -X POST \\\n",
        "    -d @request.json  2> /dev/null > response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "86c560e0a641"
      },
      "source": [
        "With the new settings, the `blocked_reason` is no longer set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 147,
      "metadata": {
        "id": "0c2847c49262"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{\n",
            "  \"safetyRatings\": [\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "      \"probability\": \"MEDIUM\"\n",
            "    },\n",
            "    {\n",
            "      \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "      \"probability\": \"NEGLIGIBLE\"\n",
            "    }\n",
            "  ]\n",
            "}\n"
          ]
        }
      ],
      "source": [
        "%%bash \n",
        "\n",
        "jq .promptFeedback < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "47298a4eef40"
      },
      "source": [
        "And a candidate response is returned."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "028febe8df68"
      },
      "outputs": [],
      "source": [
        "%%bash \n",
        "\n",
        "jq .candidates[0].content.parts[].text < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ujVlQoC43N3B"
      },
      "source": [
        "You can check `response.text` for the response."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d401c247957"
      },
      "source": [
        "### Candidate ratings"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3d306960dffb"
      },
      "source": [
        "For a prompt that is not blocked, the response object contains a list of `candidate` objects (just 1 for now). Each candidate includes a `finish_reason`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 157,
      "metadata": {
        "id": "e49b53f69a2c"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "\"STOP\"\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "jq .candidates[0].finishReason < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "badddf10089b"
      },
      "source": [
        "`FinishReason.STOP` means that the model finished its output normally.\n",
        "\n",
        "`FinishReason.SAFETY` means the candidate's `safety_ratings` exceeded the request's `safety_settings` threshold."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 158,
      "metadata": {
        "id": "2b60d9f96af0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_HATE_SPEECH\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_HARASSMENT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  },\n",
            "  {\n",
            "    \"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\n",
            "    \"probability\": \"NEGLIGIBLE\"\n",
            "  }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "%%bash\n",
        "jq .candidates[0].safetyRatings < response.json"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1UdbxVt3ysY"
      },
      "source": [
        "## Learning more\n",
        "\n",
        "Learn more with these articles on [safety guidance](https://ai.google.dev/docs/safety_guidance) and [safety settings](https://ai.google.dev/docs/safety_setting_gemini).\n",
        "\n",
        "## Useful API references\n",
        "\n",
        "- Safety settings can be set in the [genai.GenerativeModel](https://ai.google.dev/api/python/google/generativeai/GenerativeModel) constructor. They can also be passed on each request to [GenerativeModel.generate_content](https://ai.google.dev/api/python/google/generativeai/GenerativeModel#generate_content) or [ChatSession.send_message](https://ai.google.dev/api/python/google/generativeai/ChatSession?hl=en#send_message).\n",
        "- The [genai.GenerateContentResponse](https://ai.google.dev/api/python/google/ai/generativelanguage/GenerateContentResponse) returns [SafetyRatings](https://ai.google.dev/api/python/google/ai/generativelanguage/SafetyRating) for the prompt in the [GenerateContentResponse.prompt_feedback](https://ai.google.dev/api/python/google/ai/generativelanguage/GenerateContentResponse/PromptFeedback), and for each [Candidate](https://ai.google.dev/api/python/google/ai/generativelanguage/Candidate) in the `safety_ratings` attribute.\n",
        "- A [glm.SafetySetting](https://ai.google.dev/api/python/google/ai/generativelanguage/SafetySetting)  contains: [glm.HarmCategory](https://ai.google.dev/api/python/google/ai/generativelanguage/HarmCategory) and a [glm.HarmBlockThreshold](https://ai.google.dev/api/python/google/generativeai/types/HarmBlockThreshold)\n",
        "- A [glm.SafetyRating](https://ai.google.dev/api/python/google/ai/generativelanguage/SafetyRating) contains a [HarmCategory](https://ai.google.dev/api/python/google/ai/generativelanguage/HarmCategory) and a [HarmProbability](https://ai.google.dev/api/python/google/generativeai/types/HarmProbability)\n",
        "- The [glm.HarmCategory](https://ai.google.dev/api/python/google/ai/generativelanguage/HarmCategory) enum includes both the categories for PaLM and Gemini models. The values allowed for Gemini models are `[7,8,9,10]`: `[HARM_CATEGORY_HARASSMENT, HARM_CATEGORY_HATE_SPEECH, HARM_CATEGORY_SEXUALLY_EXPLICIT, HARM_CATEGORY_DANGEROUS_CONTENT]`.\n",
        "- When specifying enum values the SDK will accept the enum values themselves, or their integer or string representations. The SKD will also accept abbreviated string representations: `[\"HARM_CATEGORY_DANGEROUS_CONTENT\", \"DANGEROUS_CONTENT\", \"DANGEROUS\"]` are all valid. Strings are case insensitive.\n"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Safety_REST.ipynb",
      "toc_visible": true
    },
    "google": {
      "image_path": "/static/site-assets/images/docs/logo-python.svg",
      "keywords": [
        "examples",
        "gemini",
        "beginner",
        "googleai",
        "quickstart",
        "python",
        "text",
        "chat",
        "vision",
        "embed"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
